Towards Automated Exploration

The primary goal of auto-explore is to to establish a codebase that reduces the effort to produce a reasonable first-pass exploratory data analysis for a variety of dataset types.

This Python library is a first attempt at automating the process of exploratory data analysis – at least as far as computation and visualization is concerned.

Critical thinking is not included.

Potential Benefits of Semi-automated EDA

• Faster time to insights & modeling
• Shorter exploratory data analysis turnaround
• Reliable processes that are vetted & improved over time
• No need to re-configure old code to new situations
• Supplies a base for more in-depth analysis

Previous Work in the Space

Linear Regression Analysis

Text Analysis

Time Series Analysis

Correlation Analysis

Correlation Analysis

Categorical Analysis

Clustering Analysis

Clustering Analysis

Clustering Analysis

Helper Functions

In [10]: from auto_explore.datetime import make_calendars
In [11]: year_list = np.arange(2011, 2020)
In [12]: cal_df = make_calendars(year_list, drop_index=False)
In [13]: cal_df.head()                                                                                                          
Out[13]: 
              month  year    weekday  is_weekday  is_holiday  is_holiday_week
DATE                                                                         
2011-01-01  January  2011   Saturday           0           0                0
2011-01-02  January  2011     Sunday           0           0                0
2011-01-03  January  2011     Monday           1           0                0
2011-01-04  January  2011    Tuesday           1           0                0
2011-01-05  January  2011  Wednesday           1           0                0

Helper Functions

In [10]: from auto_explore.apis import fetch_fred_data
In [11]: series_list = ['SP500', 'NASDAQCOM', 'DJIA', 'RU2000PR'] # cboe energy sector etf volatility
In [12]: econ_df = fetch_fred_data(series_list)
In [13]: econ_df.head()
Out[13]: 
              SP500  NASDAQCOM      DJIA  RU2000PR
DATE                                              
2011-01-03  1271.87    2691.52  11670.75   1984.61
2011-01-04  1270.20    2681.25  11691.18   1952.99
2011-01-05  1276.56    2702.20  11722.89   1976.01
2011-01-06  1273.85    2709.89  11697.31   1966.88
2011-01-07  1271.50    2703.17  11674.76   1957.96            0

eda.py: Interface to Semi-Automation

Simply specify a DataFrame and a list for each of its binary, categorical, numerical and text columns. If applicable, set the target_col as a list with one element (string).

from auto_explore.eda import AutopilotExploratoryAnalysis

args = (df, bin_cols, cat_cols, num_cols, text_cols)
kwargs = dict(target_col=target_col)
ax = AutopilotExploratoryAnalysis(*args, **kwargs)

This object makes available 17 methods for use on the df supplied as an arg. Check out the most recent code on Github.

Other Modules

viz.py

featexp.py

apis.py

notebooks.py

stats.py

datetime.py